You might have worked on some simple linear regression using ordinary least squares, and its more general regression of polynomial functions. You've also seen how we can overfit models to data using polynomials and interactions. In this blog post, I want to take a look at another way to tune our linear regression models. These methods all modify the mean squared error function that you are optimizing against. The modifications will add a penalty for large coefficient weights in the resulting model. Ridge and Lasso regression are some of the simple techniques to reduce model complexity and prevent over-fitting.

Ordinary Least Squares is one of the easiest and most widely used ML algorithms. But it suffers from a fatal flaw -- it is super easy for the algorithm to overfit the training data. But as the number of predictor variables (or dimensions) increases, the coefficients β_i also tend to get very large. With large coefficients, it is easy to predict nearly everything -- you just take the relevant combination of individual slopes (βs) and you get the answer. That's why it is common for linear regression models to overfit the training data.

This article was published as a part of the Data Science Blogathon. Many times we have come across this statement – Lasso regression causes sparsity while Ridge regression doesn't! But I'm pretty sure that most of us might not have understood how exactly this works. Let's try to understand this using calculus. First, let's understand what sparsity is.

When we talk about Regression, we often end up discussing Linear and Logistics Regression. Do you know there are 7 types of Regressions? Linear and logistic regression is just the most loved members from the family of regressions. Last week, I saw a recorded talk at NYC Data Science Academy from Owen Zhang, current Kaggle rank 3 and Chief Product Officer at DataRobot. He said, 'if you are using regression without regularization, you have to be very special!'. I hope you get what a person of his stature referred to. I understood it very well and decided to explore regularization techniques in detail. In this article, I have explained the complex science behind'Ridge Regression' and'Lasso Regression' which are the most fundamental regularization techniques, sadly still not used by many.

When we talk about Regression, we often end up discussing Linear and Logistics Regression. Do you know there are 7 types of Regressions? Linear and logistic regression is just the most loved members from the family of regressions. Last week, I saw a recorded talk at NYC Data Science Academy from Owen Zhang, current Kaggle rank 3 and Chief Product Officer at DataRobot. He said, 'if you are using regression without regularization, you have to be very special!'. I hope you get what a person of his stature referred to. I understood it very well and decided to explore regularization techniques in detail. In this article, I have explained the complex science behind'Ridge Regression' and'Lasso Regression' which are the most fundamental regularization techniques, sadly still not used by many.